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A  Dynamic  Mission  Replanning  Testbed  for 
Supervisory  Control  of  Multiple  Unmanned  Aerial  Vehicles 

Jeremy  Nelson,  Gloria  Calhoun,  &  Mark  Draper 
Abstract 

As  unmanned  aerial  vehicles  (UAVs)  increase  in  autonomy,  operators  will  be 
increasing  their  span  of  control.  Most  UAV  systems  require  two  or  more  operators  to  fly 
and  operate  payloads,  but  systems  are  being  developed  with  the  concept  of  a  single 
operator  monitoring  multiple  UAVs.  This  supervisory  control  of  multiple  UAVs  raises 
many  issues  concerning  the  balance  of  system  autonomy  with  human  interaction  to  keep 
the  operator  in-the-loop.  Testbeds  are  needed  that  specifically  address  multi-UAV 
supervisory  control,  replicating  the  complex  automation  algorithms  and  allowing  operator 
initiation  and  inspection  into  these  systems.  There  is  currently  an  effort  underway  to 
develop  a  dynamic  mission  replanning  testbed  for  human  factors  research  on  supervisory 
control  of  multiple  UAVs.  This  testbed  utilizes  Air  Force  certified  autorouting  software 
and  creates  a  tool  to  begin  tackling  issues  many  of  these  issues.  A  preliminary  study  is 
being  performed  with  this  still  developing  testbed  and  results  will  be  presented. 

Introduction 

With  advances  in  technology,  unmanned  aerial  vehicles  (UAVs)  are  becoming 
ever  more  self-sufficient.  Some  systems  are  already  capable  of  completing  entire  pre¬ 
planned  missions  autonomously.  Therefore,  there  is  a  logical  progression  to  reduce  the 
manpower  required  to  operate  each  vehicle.  This  shift,  from  one  or  more  operators  flying 
and  operating  payloads  on  a  single  vehicle  to  one  operator  supervising  multiple  vehicles 
while  managing  other  tasks,  is  complex.  A  major  challenge  is  in  determining  the  optimal 
involvement  of  the  human  operator  at  all  times  in  all  situations  -  how  much  of  each  task 
should  be  accomplished  by  the  operator  and  how  much  should  be  accomplished  by  the 
automation?  Also,  the  appropriate  level  of  autonomy  will  likely  vary  from  function  to 
function  and  change  depending  on  mission  type  and  phase.  Furthermore,  with  highly 
automated  UAV  systems,  additional  human  factors  issues  to  be  considered  include 
human  vigilance  decrements,  “clumsy  automation”,  limited  system  flexibility,  mode 
awareness,  trust/acceptance  issues,  failure  detection,  automation  biases,  etc. 

(Parasuraman  &  Mouloua,  1996;  Gawron  &  Draper,  2001). 

Research  addressing  the  ideal  balance  of  task  allocation  between  the  operator  and 
UAV  automated  systems  can  benefit  from  testbeds  designed  to  emulate  the  supervisory 
requirements  of  multi-UAV  control.  Some  current  testbeds  include:  UAV  Modeling  And 
Simulation  Testbed  (UMAST;  Ruff,  Narayanan,  &  Draper,  2002);  Multi-Modal 
Immersive  and  Intelligent  Interface  for  Remote  Operation  (MIIIRO;  Ruff,  Calhoun, 
Draper,  Fontejon,  &  Guilfoos,  2004);  Tactical  Tomahawk  Interface  for  Monitoring  and 
Retargeting  (TTIMR;  Cummings,  2003);  and,  Multi- Aerial  Unmanned  Vehicle 
Experiment  (MAUVE)  interface  (Cummings  &  Mitchell,  in  review).  While  each  testbed 
has  its  strengths  and  weaknesses  in  terms  of  simulation  fidelity,  it  is  debatable  whether 
current  testbeds  have  sufficient  underlying  mission  planning  fidelity/complexity  to 
adequately  represent  the  supervisory  control  requirements  anticipated  for  future  multi- 
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UAV  operations.  This  paper  will  explore  testbed  design  requirements  for  multi-UAV 
applications  as  well  as  describe  a  new  testbed  under  evaluation. 


UAV  Dynamic  Mission  Replanning:  Human  Factors  Concerns 

Dynamic  mission  replanning  systems  contain  automation  algorithms  which  allow 
missions  to  be  modified  in  real-time.  These  modifications  may  occur  in  several  main 
mission  areas:  allocation  of  tasks  to  vehicles;  route  replanning,  sensor  management, 
communications  planning,  and  weapons  management.  Most  of  the  current  testbeds  lack 
the  complexity  in  automation  and  range  of  autonomy  levels  for  multiple  mission  tasks. 

In  one  study  utilizing  the  MIIIRO  testbed,  for  example,  the  route  replanning  automation 
logic  was  relatively  simple  and  the  human  interaction  with  it  required  minimal  cognitive 
effort  (i.e.,  does  the  new  route  cross  another  UAV’s  route  or  enter  a  threat  ring?).  In  the 
experiment,  participants  rushed  to  manually  respond  fast  enough  to  avoid  having  the 
system  automatically  respond.  Thus,  for  this  experimental  paradigm,  automation  was  not 
a  workload  reducer  (Ruff  et  al.,  2004).  The  problem  was  not  with  the  testbed  per  se,  but 
with  the  complexity  of  the  automation.  This  will  not  suffice  for  experimentation,  as  route 
replanners  will  need  to  process  many  complex  and  abstract  variables  simultaneously  in 
order  to  generate  new  routes. 

Future  UAV  operators  must  interact  with  highly  complex  systems,  understanding 
critical  pieces  of  information  such  as  why  the  system  chose  a  specific  action,  what 
criteria  it  used  in  making  the  decision,  and  what,  if  any,  tweaking  can  be  made  to  enhance 
mission  success.  The  UAV  operator’s  cognitive  resources  will  be  required  to  inspect  the 
automation  because  algorithms  operate  on  assumptions  and  will,  at  times,  contain 
idiosyncrasies  or  have  an  incorrect  model  of  the  world  state.  With  netcentric  feeds  and 
intelligent  agents  providing  large  amounts  of  data  to  the  system,  operators  will  be 
required  to  drill  into  the  automation  algorithms  to  investigate  issues  such  as  accuracy  and 
recency  of  data  used  for  decisions.  However,  if  the  interface  to  accomplish  this 
inspection  is  not  human  factored,  workload  will  increase.  Additionally,  inspection 
methods  for  these  complex  algorithms  may  not  be  straightforward,  dependent  upon  the 
procedures  used  to  ascertain  new  routes  and  task  assignments.  It  is  also  important  that 
the  operator  maintain  overall  situation  awareness  and  attention  not  be  tunneled  on  a 
specific  inspection  task.  Therefore,  researchers  need  to  explore  how  best  to  design  the 
interface  by  which  an  operator  monitors  and  inspects  automation  for  a  multi-UAV 
system. 

Though  future  UAV  operators  will  interact  with  automation  in  a  variety  of  ways, 
two  of  the  most  important  will  be  the  allocation  of  tasks  among  multiple  UAVs  and 
maneuvering  of  UAVs  to  achieve  mission  objectives.  Nearly  all  events  that  occur  in  a 
multi-UAV  environment  will  have  an  impact  on  task  allocation  and  the  routing  of  the 
aircraft,  such  as  new  threats  or  targets,  changes  in  rules  of  engagement,  changes  in  health 
and  status  of  a  vehicle,  communication  links,  and  weather.  Moreover,  the  system  must 
coordinate  route  planning  with  sensor  requirements,  ingress/egress  paths,  aircraft 
deconfliction,  and  pre-defined  aircraft  operating  areas  and  altitudes.  These  complexities 
require  a  testbed  that  more  fully  replicates  dynamic  mission  replanning.  An  effort  is 
underway  to  develop  a  testbed  that  enables  user  interaction  with  a  high  fidelity  dynamic 
mission  replanning  system  and  employs  adaptive  levels  of  automation. 
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The  ALOA  Testbed 

The  Adaptive  Levels  of  Automation  (ALOA)  testbed  is  a  multi-UAV  mission 
control  station  emulator  (MCSE)  being  developed  as  part  of  the  Air  Force  Research 
Laboratory’s  program  to  address  human  factors  associated  with  multi-UAV  operator 
control  stations.  This  UAV  operator/automation  research  testbed  is  unique  in  that  it 
incorporates  a  commercially  available  dynamic  mission  replanner,  increasing  the 
testbed’s  complexity  and  realism  (Johnson,  Leen,  Goldberg,  &  Chiu,  2005). 

The  backbone  for  the  ALOA  MCSE  testbed  is  Operations  Research  Concepts 
Applied’s  (ORCA)  OPUS  software.  Used  for  mission  planning,  OPUS  specializes  in 
autorouting  tools  for  low-observable  aircraft.  OPUS  is  certified  for  operational  use  by 
the  Air  Force  and  endorsed  by  the  Navy’s  UAV  Advanced  Technology  Review  Board. 

By  means  of  vehicle-threat  interaction  models,  OPUS  takes  into  account  vehicle  radar 
cross  section  data  and  radar  vertical  coverage  diagram  data.  These  tools  and  models 
provide  the  ALOA  MSCE  testbed  with  a  realistic  operational  mission  planning 
environment. 

Besides  its  high  fidelity  routing  abilities,  this  testbed  will  also  incorporate  both 
adaptable  and  adaptive  levels  of  automation  for  a  variety  of  operator  tasks.  Through 
adaptable  automation,  operators  actively  change  the  current  level  of  system  autonomy  for 
each  task.  The  adaptable  automation  can  also  be  grouped  into  predefined  autonomy 
levels  for  each  task,  much  like  a  stereo  equalizer,  allowing  an  operator  to  pick  a  broad 
“scheme”  autonomy.  However,  the  task  of  changing  autonomy  levels  can  quickly 
become  an  additional  burdensome  task  for  the  operator.  One  solution  is  to  combine  the 
components  of  an  adaptable  system  with  those  of  an  adaptive  system.  In  adaptive 
automation,  the  system  initiates  changes  in  the  autonomy  level  based  on:  1)  real-time 
measurement  of  the  operator’s  performance  on  tasks;  2)  current  mission  profile  and 
sensed  events;  and,  3)  the  operator’s  physiological  measurements.  The  ALOA  MSCE 
plans  to  incorporate  all  of  these  controls  and  allow  for  plug-in  physiological  metric 
inputs.  This  will  enable  the  examination  of  how  adaptive  automation  (the  computer 
actively  changing  autonomy  levels)  can  help  the  operator  maintain  a  manageable 
workload  level,  while  preserving  his/her  situation  awareness. 

The  ALOA  system  is  still  undergoing  development.  However,  an  experiment  is 
currently  underway  with  an  early  version  of  the  system  to  obtain  a  preliminary 
assessment  of  autonomy  levels  in  an  operator  route  selection  task,  as  well  as  to  evaluate 
the  functioning  of  the  ALOA  testbed  and  identify  needed  refinements. 

Preliminary  ALOA  Study 

This  study  is  manipulating  the  number  of  UAVs  under  control  (1,  3,  or  4)  and 
three  autonomy  levels  of  the  auto-routing.  In  the  multiple  options  autonomy  level,  the 
participants  are  presented  with  the  current  route  and  two  modified  routes.  The  consent 
autonomy  level  presents  the  operator  with  a  single  route  but  requires  operator  consent 
before  implementing.  The  automatic  with  feedback  autonomy  level  notifies  the  operator 
after  the  computer  has  selected  and  implemented  a  route  (but  does  not  allow  a  veto 
option).  Using  a  within-subjects  design,  each  participant  completes  18  randomized  eight- 
minute  trials  (3  UAV  levels  X  3  autonomy  levels  X  2  repetitions). 
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Figure  1.  ALOA’s  dual  screen  layout. 


The  ALOA  testbed  utilizes  a  dual  screen  layout  (see  Figure  1).  Participants  use 
the  left  monitor  to  perform  image  identification  and  weapon  release  tasks,  monitor  chat, 
allocate  tasks,  replan  routes,  and  access  vehicle  data  such  as  current  location  coordinates. 
The  right  monitor  contains  the  tactical  situation  display,  health  and  status  display,  current 
route  information,  and  other  response  buttons.  While  monitoring  one  or  more  vehicles 
performing  a  suppression  of  enemy  air  defense  (SEAD)  mission,  participants  perform 
several  different  tasks:  respond  to  unidentified  aircraft;  replan  vehicles  when  new  threats 
appear;  monitor  the  health  and  status  of  vehicles;  identify  incoming  images;  and,  answer 
Situation  Awareness  Global  Assessment  Technique  (SAGAT)  questions  when  the 
simulation  is  paused.  This  experiment  is  underway,  and  full  results  will  be  presented  at 
the  conference. 


Future  Research  Directions 

There  exist  a  myriad  of  issues  surrounding  operator  inspection  of  complex  routing 
and  task  assignment  algorithms.  For  example,  dynamic  replanning  can  be  very  time 
dependent.  The  amount  of  time  available  to  react  will  dictate  the  type  of  replanning 
initiation  and  inspection  that  is  possible.  Short-reaction  threats  may  require  automatic 
evasive  maneuvers  and  replans.  Over  longer  periods,  more  operator  involvement  in 
replan  inspection  is  possible,  but  it  has  yet  to  be  determined  how  beneficial  human 
intervention  will  be.  Design  factors  such  as  the  amount  of  time  given  to  the  operator  to 
replan,  what  information  should  be  most  readily  available,  and  how  the  automation 
should  play  into  those  replans  need  to  be  examined. 

Future  studies  will  also  be  required  to  examine  automation  bias,  adaptive  and 
adaptable  levels  of  automation,  automation  schemes,  and  scenarios  with  changing  rules 
of  engagement.  While  the  ability  to  study  unreliable  automation  in  the  routing  is  possible 
with  the  ALOA  simulator,  the  concept  of  reliable  automation  lacking  all  necessary 
information  is  a  particular  area  of  interest.  This  will  be  accomplished  by  changes  in  the 
rules  of  engagement  through  the  chat  window.  For  example,  the  routing  software  will 
calculate  what  it  considers  the  best  route  based  on  the  given  parameters  for  fuel  economy, 
threat  avoidance,  etc.  However,  based  on  the  most  recent  rules  of  engagement,  the 
operator  has  been  directed  to  strike  a  target  regardless  of  threat  exposure.  In  this  case,  the 
automation  is  not  incorrect  but  simply  lacking  in  all  the  latest  contextual  information. 

The  automation  is  only  as  good  as  the  information  it  can  access. 
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Due  to  the  complexity  of  the  automation  algorithms,  it  is  difficult  to  decide  which 
inputs  to  display  to  the  operator.  Autorouting,  for  instance,  considers  vehicle 
performance,  threat  susceptibility,  and  terrain  parameters  to  determine  a  feasible  route 
that  achieves  mission  objectives.  Within  those  considerations,  there  are  numerous 
parameters  that  can  be  adjusted  to  alter  the  output.  Breaking  down  which  components  to 
display  to  the  operator  and  which  to  allow  to  be  manipulated  are  other  questions  that 
require  further  research.  With  the  flexibility  of  the  ALOA  MSCE  testbed,  it  will  be 
possible  to  explore  issues  such  as  these. 
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